Software Testing – Redefined

By: R. Sankara Narayanan

Part:   1  2   3 

(Continued from previous part...)

Refer Table 2. This matrix is prepared for a website to validate the registration page and its drop-down box elements. This matrix also contains the list of Page + Field details across the rows (row headers) and the list of actions to be performed with drop-down box elements, across the columns (column headers).

The above stated matrix methodology need not to be restricted to field validations and page validations, they may get extended to any repetitive set of actions to be performed with many elements in the product. A skilled explorative tester would be able to create these kinds of matrices and apply them in the process of software testing.

Unscripted Testing - Exploratory Testing without Scripted Tools

As the name implies, Unscripted Testing is the testing that doesn’t uses any scripted tools. There are different names in practice, for this testing – Monkey Testing, Adhoc Testing, Guerilla Testing, etc…. The practice of smoke testing, if it is unscripted, falls under this category only.

As pointed out earlier, the exploratory testing also falls under this category. An explorative tester requires checklists/matrices just to facilitate the process of testing in order wherein no detail is missed. However, that is not always a mandatory requirement. A skilled explorative tester may also be able to prepare these checklists/matrices and retain them in his/her mind. Exploratory Testing is something similar to playing blind chess. In blind chess, the players are not able to see the pieces but they can conceptualize where the pieces are located intellectually. Similarly, the exploratory testers may not have any scripted tools but may conceptualize what they are supposed to test, intellectually in a well-planned manner.

Exploratory Testing with scripted tools is something similar to riding a motor bike/car in a narrow city road. The tester has a way; a well designed scripted methodology and can reach the destination perfectly. But there is no chance to explore other parts of the city, if one is only driving on the roads.

On the other hand unscripted exploratory testing is something similar to shipping in Indian Ocean for the purpose of fishing. The teser is subjected to unrestricted area to cover. In order to catch the fishes, the fisher man does follow a standard and proven methodology like, “throw the net - waiting for some time for the fishes to gather in the area, covered by the net”. Similarly in order to uncover software product defects, an exploratory tester follows a standard and proven methodology like study the requirements/use cases, design a strategy and execute them simultaneously though in unrestricted realms of product design.

With Test Cases – What do I miss?

The task of creating test cases includes writing, reviewing and rectifying it according to the review defects. It is obvious that reviews are effective and costly. Also a very significant amount of time is invested in creating test cases.

If there is no requirement for Test Repeatability, Portability & Legality, and if the test manager chooses to go for creating the test cases, the time/cost that is invested in such an effort would not be justified. This is because, in such a case, the test case lifecycle would be very small. To begin with more than 50% of the total test effort would be spent towards creating the test cases. The test cases once created would be executed only once/twice. Thus we see test cases thrown away after 100% of the total test effort, which is not prudent in any given situation.

If “Use and Throw” is the requirement of test tools, checklists/matrices and all categories of requirement specifications would be the best bet to get the inputs for testing. The time that is required for the preparation of the checklists/matrices is very negligible when compared to creating test cases. Also they need not be maintained in an electronic format like word document, excel document or a database. They can be made available in a piece of paper, chip paper, or notepad allowing tester to just use it and throw.

Test Planning can be met not only by the test cases but also with all other alternatives as mentioned in the previous section. Just to meet this objective, a test manager need not choose creating test cases.



Test Case Automation – Why, When & How

Automation helps overcome the disadvantages of Manual Regression Testing. Repeatability, Reliability and Execution Speed are the critical factors that recommend Test Automation. Creation and Maintenance of Automated Test Scripts needs subsequent amount of effort. Careful planning is key to succeed in the Test Automation process.

Section 15 of STS’s SSTP V1.2 states that a set of test cases shall get identified as a candidate for automation if and only if it warrants multiple rounds of execution where estimated cost of automation is less than the estimated cost that will not be incurred upon automation.

How to interpret this section 15? “Estimated Cost of Automation” is the total effort that is estimated towards the task of automating the identified set of test cases for the purpose of test automation. “Estimated Cost that would not be incurred upon automation” is the estimated cost of manual effort that would not be incurred upon automation.

Say, a set of test cases takes 5 person days for one-round-execution and upon automation takes 2 person days for one-round-execution. 3 person days would not be needed for one-round-execution after automating the test cases. If the task of test automation takes 15 days, the scope of test automation project shall have at least 15/3 = 5 rounds of execution so that the investment would not suffer a loss. For 5 rounds of testing, the estimated cost that would not be incurred upon automation is the cost that is associated with 15 person days. If the project has more than 5 rounds of testing, for each round of testing, the cost towards 3 person days would not be incurred and this enhances the productivity of the testing.

To make the interpretation more complex, say, there are 3 sets of test cases each taking 5 person days for one-round-execution and upon automation each taking 1 person days for one-round-execution. If these 3 sets of test cases were run in sequence, it would generally take 3 person days for one-round-execution. If these 3 sets of test cases were run in parallel, it would generally take 1 person day for one-round-testing (the max execution time of the 3 sets of test cases). In this case, the estimated cost of the manual effort that would not be incurred upon automation would be
= (3*5) – max (1,1,1)
= 15 – 1
= 14 person days.


Note: Parallel Run of Automated Scripts should also be considered for calculating the return of investment of Test Automation.

Test Planning

Following the above discussion on the various testing methodologies here is the summation of how one can apply the given guidelines to form an effective test plan.

According to Section 9 and 10 of STS’s SSTP V1.2, if the test repeatability, portability and legality are irrelevant, the test manager will not opt for creating and maintaining test cases but will opt only for exploratory testing – scripted or unscripted. In this situation, the following questions arise:

Practically, is it feasible to have only exploratory testing?
Practically, is it acceptable?


There are fewer situations where the test repeatability, portability and legality are not required to be met. In most of the situations, test repeatability, portability and legality are required to be met. The questions now get transformed into the following question:

Up to what extent these objectives shall be met?

Practically, the answer will not be 0% or 100% but it lies somewhere between 20 – 80% on a subjective scale. The next questions would be:

How to plan for that?

Apply Section 9 and 10 of STS’s SSTP V1.2.

When do I go for test automation?

Interpret Section 15 of STS’s SSTP V1.2. Apply it.

(Continued on next part...)

Part:   1  2   3 

Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/